A task level fusion autonomous switching mechanism

Positioning technology is an important component of environmental perception. It is also the basis for autonomous decision-making and motion control of firefighting robots. However, some issues such as positioning in indoor scenarios still remain inherent challenges. The positioning accuracy of the fire emergency reaction dispatching (FERD) system is far from adequate to support some applications for firefighting and rescue in indoor scenarios with multiple obstacles. To solve this problem, this paper proposes a fusion module based on the Blackboard architecture. This module aims to improve the positioning accuracy of a single sensor of the unmanned vehicles within the FERD system. To reduce the risk of autonomous decision-making of the unmanned vehicles, this module uses a comprehensive manner of multiple channels to complement or correct the positioning of the firefighting robots. Specifically, this module has been developed to fusion a variety of relevant processes for precise positioning. This process mainly includes six strategies. These strategies are the denoising, spatial alignment, confidence degree update, observation filtering, data fusion, and fusion decision. These strategies merge with the current scenarios-related parameter data, empirical data on sensor errors, and information to form a series of norms. This paper then proceeds to gain experience data with the confidence degree, error of different sensors, and timeliness of this module by training in an indoor scenario with multiple obstacles. This process is from data of multiple sensors (bottom-level) to control decisions knowledge-based (up-level). This process can obtain globally optimal positioning results. Finally, this paper evaluates the performance of this fusion module for the FERD system. The experimental results show that this fusion module can effectively improve positioning accuracy in an indoor scenario with multiple obstacles. Code is available at https://github.com/lvbingyu-zeze/gopath/tree/master.


Introduction
With the rapid development of robots, they are used in various fields [1].Researchers have focused on firefighting robots that have reduced casualties among rescuers.When a fire occurs in an indoor environment, the firefighting robot can adjust to the dynamic indoor environment in order to achieve a safe rescue or accurate firefighting [2,3].Accurate positioning is very important for firefighting robots in order to be able to do rescue work accurately [4].In indoor scenarios where firefighting robots perform tasks, GPS signals are often unavailable [5].The current indoor positioning methods have many methods, including ultra wide band (UWB), inertial measurement unit (IMU), infrared depth sensor (IDS), camera [4,[6][7][8][9][10][11]. UWB has high temporal resolution but is susceptible to indoor environments [4,6].The IMU is highly accurate, but there is a cumulative error [7].IDS is easily detected at close range but has limited detection distance, high noise, and low accuracy [8,9].Camera accuracy is great, but the real-time performance is poor and vulnerable to light conditions [10,11].These sensors can be used to their advantage in different usage scenarios and are difficult to replace.Therefore, the problem of indoor positioning is still inherent.
The single sensor has limited information and is vulnerable to environmental influences.As a result, firefighting robots usually carry a large number of different types of sensors.The information acquired by multiple sensors may complement or contradict each other [12,13].The information acquired by each sensor has to be processed separately, increasing the workload.Therefore, multi-sensor fusion is necessary to process the information.Information fusion can improve information reliability, robustness, and utilization [4,5,7].The existing research on multi-sensor fusion framework mainly focuses on the response to specific situations [7,14,15].Sara et al. [14] proposed a three-level positioning framework based on additional visual sensors.The framework obtains the absolute position of the vehicle by tracking landmarks and also improves the GNSS measurement method.Li et al. [7] designed an indoor positioning method based on multiple sensors.They also constructed an extended Kalman filtering (EKF) algorithm framework.Dasgupta et al. [15] proposed sensor fusion software for autonomous vehicles and other autonomous robots to achieve precise positioning of autonomous robots.However, multi-sensor fusion is susceptible to environmental interference, leading to measurement errors and uncertainties.Therefore, facing a large amount of uncertain data, multiple domain experts are needed to process this data.The blackboard architecture has proven to be a very useful and flexible mechanism [16,17].In this paper, a fusion module in FERD system is proposed based on the Blackboard Architecture.The overall goal is achieved by quickly and autonomously switching different strategies for multiple unmanned vehicles.The position of unmanned vehicles is supplemented or modified by multiple channels.These strategies can reduce the impact of inaccurate positioning in an open environment, also reduce the risk of autonomous decision-making of each unmanned vehicle.Then, this fusion module in FERD system is realized in the task of firefighting.Finally, the performance of the proposed fusion module is evaluated.The main contributions of this paper are summarized as follows.
• In this paper, a fusion module is designed and added to fire emergency reaction dispatching (FERD) system.This fusion module is used for accurately positioning fire equipment by multiple sensors.All sensors send their measured position information as primary data to this module.The position information of the real physical position of each fire equipment is used as auxiliary data.This module has developed a variety of fusion-related processing for precise positioning.These processes mainly include six strategies.They are the denoising, spatial alignment, confidence degree update, observation filtering, data fusion, and fusion decision.Compared to a single sensor, this paper proposed module is effective and scalable.
• Multiple unmanned vehicles perform firefighting tasks and collect data in an indoor scenario with multiple obstacles.This demonstrates the effectiveness of the module in locating sensors in the field.This module merges with the current scenarios-related parameter data, empirical data on sensor errors, and information to form a series of norms.This paper then proceeds to gain experience data with the confidence degree, error of different sensors, and timeliness of this module by training in an indoor scenario with multiple obstacles.This process is from data of multiple sensors (bottom-level) to control decisions knowledge-based (up-level).This process can obtain globally optimal positioning results.
The remainder of this paper is organized as follows.Section RELATED WORKS introduces the related to the work of this paper.Section PRELIMINARIES introduces a few concepts and reviews previous works.Then, a few basic terms are provided that will be used in later sections.Section PROPOSED MODULE, this section emphasizes proposed module.Section RESULTS analyzes the errors of different sensors in detail.Then, this section introduces how to implement the fusion module based on Blackboard Architecture.Finally, this section gives the experimental results and analysis.Section DISCUSSION discusses the advantages and disadvantages of the proposed fusion module.Section CONCLUSION summarizes the significance of the fusion module and discusses future work.

Related works
This paper presents a novel and comprehensive sensor fusion scheme.The focus of this work is on the fusion method and the switching of fusion methods.

Kalman filtering
Some existing works with us have related ideas to Kalman Filtering as one of the data fusion methods.For the indoor positioning problem of miniature tracked robots, Li et al. [7] designed an indoor positioning method.This method is based on Bluetooth, a gyroscope, an accelerometer, a magnetometer, and other sensors.This method takes an inertial navigation system (INS) as the core, Bluetooth AOA positioning base station as the position observation, and magnetometer as the heading angle observation.They also constructed an EKF algorithm framework.This EKF is based on error states.Kim and Lee [18]proposed the EKF algorithm.This algorithm is a combination of a Camera, GPS, and sensor of in-vehicle for the precise positioning of vehicles.This algorithm combines multiple decisions made by different sensors and finally makes a fusion decision.Luo et al. [19] proposed a joint Kalman filter to fuse the positioning information.This information is the combined localization subsystem.This method provides critical position information for achieving high accuracy and efficient missions.The work of this paper determines whether to fuse the data from multiple sensors through confidence degree and error size.Then this work of this paper uses the Kalman filter to fuse the data.In this way, the proposed approach provides critical position information during operation.The proposed approach improves the positioning accuracy and efficient tasks of FERD system.

Multi-sensor fusion based blackboard architecture
The proposed fusion module combines the norms of data fusion and error parameters of multiple sensors with the Blackboard Architecture.Benjamin et al. [20] proposed that the design of the transmission trigger and estimator should be closely combined.The intelligent trigger mechanism of sensors can predict future information about these sensors.The purpose of this method further reduces the communication demand.Girraj et al. [21] optimized and evaluated thresholds of fusion norms.They proposed an iterative algorithm to calculate the individual optimal threshold of fusion norms.Based on the necessary and sufficient conditions of the optimal sensor norm, Liao et al. [22] derived a set of norms for sensors.These norms are significantly superior to the traditional iterative search algorithm.These norms are applied to multi-sensor decision fusion problems with high-dimensional sensors.Yang et al. [23] proposes a sensor fusion algorithm.They create redundancy, by using multiple sensors to measure the same physical variables.This algorithm is used to provide robust estimation with correct sensor information.These controllers reduce estimation errors and communication channel noise.However, multi-sensor fusion is susceptible to environmental interference, leading to measurement errors and uncertainties.The difference between the method proposed in this paper and these methods is that the accuracy of multiple sensors is considered, and some norms about accuracy are designed for fusion.In different environments, our method can filter the measurement accuracy of multiple sensors and switch the corresponding strategies about fusion.Since norms of accuracy require a combination of parametric and empirical data related to the current environment, firefighting robots need to be trained to obtain these data.These training values can be used as empirical values for FERD system to perform tasks.In this way, the method proposed in this paper is more likely to improve the correctness of decisions and reactions during the operation of FERD system.

Preliminaries
In this section, the Blackboard System (BBS) and General Blackboard Open Source (GBBopen) are first introduced.Then, previous work and some concepts are introduced.Finally, the basic concepts covered in this paper are introduced in detail.

Blackboard system and generic blackboard open source
(1) Blackboard system.BBS is an artificial intelligence technique based on knowledge bases and collaborative processing, which is a distributed computing model.It is widely used in complex problem solving, for example, in the field of decision support.It enables effective collaborative processing of multiple specialized modules to achieve more intelligent and efficient problem solving.The operation of the BBS is driven by events.It comprises a knowledge source (KS), blackboard (BB), and control shell (CS).In the BBS, a KS is similar to an expert.Each KS has its own expertise and capabilities.These KSs work together through a central data structure (BB) to solve a complex problem.BB is a shared data structure.Specifically, the information on the BB can be the description of the problem, the solution of the problem, etc.All KSs can read and write the information on BB to achieve the goal of solving problems together.A KS can write the problem or the solution on BB.KS can also read the information on the BB and process and solve the problem according to its knowledge and abilities.CS is an important part of the BBS.It is responsible for monitoring the information on the BB as well as coordinating and controlling the interaction and decision-making process between the various KSs.When new information appears on the BB, the CS can decide which KSs can read it and how to deliver it to other KSs according to predefined norms and strategies.
GBBopen is an open-source blackboard system.It can be used to build various types of intelligent systems and applications.GBBopen provides a range of tools and supports multiple programming languages and operating systems.It can run on different platforms and be extended and customized on demand.Developers are able to quickly build and customize their own blackboard systems, implementing various types of intelligent applications and systems.In this paper, GBBopen is developed and implemented in the common list processing (Common Lisp) language environment.An expert system based on the norm-base developed by Clips (https://www.clipsrules.net) is a data-driven program.Unlike Clips, GBBopen does not require a data abstraction design; therefore, the implementation of a function may be completed quickly during development.
In GBBopen, the unit class is the base class for all classes defined.The properties of KS are 'trigger-events', 'precondition-function' and 'execution-function'.Each knowledge source activation (KSA) is an instance of a KS.The attributes of KS and KSA as follows: ks i = {event, pre, exe, rating}; ksa i = {rating, exe}.The running process of the CS is as follows: First, enter a series of KS.When an event occurs, the CS determines whether it is consistent with the ks i .event.If they are consistent, the state reads the data of agents from BB and the function ks k .pre is executed.The variable value of ks i .pre is provided by state.If the return value of the function ks i .pre is true, then ks i is triggered and generates an instance ksa i , which stores a list of 'pending-ksas'.Then, ksa k .exe in 'pending-ksas' with the highest level is executed.The data of the agents on BB are changed.When certain conditions are met, the GBBopen ends its operation.

Autonomous switching of task-level strategies
In this subsection, we have previously developed a framework for ASTS [24].ASTS can automatically switch strategies according to the different scenarios to react to various emergencies.
Implementation for ASTS.ASTS is described in the form of tuples, a set of norms N, state S, and event E.

ASTS ¼< N; S; E > ð1Þ
• Norm N. N = {r 1 , r 2 , . ..} represents a set of norms, which are the mapping relationships between predefined conditions and agent behavior and are the key to the effective operation of ASTS.Norm r i is expressed as • Tri i is the trigger condition and is used to determine if the associated actions can be performed.Tri i consists of a series of logical judgment, which can be expressed as Tri i ¼ ftri i 1 ; tri i 2 ; . ..g.
• Act i is the execution action.Act i consists of a series of methods or actions, which can be expressed as Act i ¼ fact i 1 ; act i 2 ; . ..g.
• Exp i is the expected result to verify the result of Act i .Exp i consists of a series of expected conditions, which can be expressed as Exp i ¼ fexp i 1 ; exp i 2 ; . ..g.
• State S. The data are read by State S. State S represents a set of instantaneous states denoted by S = {s 1 , s 2 , . ..}, which is arranged in chronological order.The instantaneous state s i consists of a series of expressions, which are expressed as s i ¼ fs i 1 ; s i 2 ; . ..g.
• Event E. E = {e 1 , e 2 , . .., e i } represents a set of instantaneous events sorted in chronological order.The instantaneous event e i consists of a series of sub-events, which are expressed as e i ¼ fe i 1 ; e i 2 ; . ..g.When an event e i occurs, the instantaneous state s j is read to determine the Tri k of r k .
Operation process of ASTS.Definition 1. Norm, Event, and State.Eq (1) describes the components of ASTS.For example, a logistics truck named Tom starts work at 1o o'clock every day.Tom must follow the norms listed in the Table 1.According to norms r a and r b , Tom maintains a safe distance W from the object in front of it (set at 0.5m).If W is greater than 0.5 m and the power is greater than 30% when e 1 occurs, then Tom moves forward.To verify the Act a result of the norm r a , we expect W to be equal to 0.5 m.If W is less than 0.5 m and the power is less than 30% when e 1 occurs, Tom stops.The instantaneous events and states are listed in Table 2. Definition 2. Triggered norm.When e i occurs, the instantaneous state s i is read.If all logical judgment expressions in Tri x are satisfied, then Norm r x is denoted as the 'triggered norm' at instantaneous state s i .For example, an instantaneous event e 1 occurs at 9:00.Tri a is satisfied at the instantaneous state s 1 ; then Tom starts moving forward.Norm r a is denoted as the 'triggered norm'.Definition 3. State transition.When e i occurs, norm r x is triggered at s i , and Act x is executed.This process causes the instantaneous state to change from s i to s j , which can be expressed as: Where the notation '!' denotes an instantaneous state changed from s i to s j as a result of triggering r x .If the instantaneous state changed from s 1 to s 2 as a result of triggering r a .The instantaneous state changes from s 2 to s 3 as a result of triggering r a .These processes are expressed in Formula (3).
Definition 4. Strategy.Norms are the key to the effective operation of multiple agent systems and are a predefined mapping relationship between conditions and behaviors.A strategy is a plan of action based on the current environment and goals.It is used to guide the agent's decisions and behaviors in a specific domain and helps the agent achieve its goals in that domain.A strategy consists of a set of action plans, and norms can be used as part of developing a strategy.For example, Norms r a and r b form a goal-following strategy in Table 1.Norm r c forms an obstacle-avoidance strategy.
Definition 5. Switch norms or strategies.When e i occurs, norm r x is triggered at s i .Then, e j occurs and r y is triggered at s j .The norm switches from r x to r y .Multiple switching norms accumulate when one norm is switched to another.For example, when e 1 occurs, Norm r a is triggered at s 1 .Then, e 2 occurs, and the norm r a is triggered at s 2 .Finally, e 3 occurs, and the norm r b is triggered at s 3 .The norm switches from r a to r a and eventually to r b .
Based on the above description, this study summarises the pseudocode of ASTS as follows: i. Input N, S, E, T; ii. Initialisation:

Fire emergency reaction dispatching system
We previously developed a FERD system applied to a fire scenario to demonstrate the autonomy and switchability of ASTS [24].We designed five strategies and two control modes for the FERD system to enable multiple types of firefighting equipment to switch between multiple strategies and multiple control modes depending on the changing environment.The system is described in detail below.Components of FERD system.The FERD system system is based on the framework of ASTS.The components include a set of tasks M, a set of fire equipment A, a set of sensors Q, and a set of agent norms N 0 .FERD system can be expressed as: • Task M. The FERD system is applied to handle fire extinguishing tasks, Task M = {m 1 , m 2 , . ..}.Each task m i comprises a series of attributes denoted as m i ¼ fm i 1 ; m i 2 ; . ..g.
• Agent A. A certain number of agents is needed to achieve the ultimate goal, which can be expressed as A = {a 1 , a 2 , . ..}. a i denotes the i-th fire equipment.a i = {u i1 , u i2 , . .., u ik } denotes the attribute of a i .u ik denotes the k-th attribute of a i .
• Sensor Q.Each agent carries sensors to sense the environment.Sensor Q = {q 1 (τ 1 ), q 2 (τ 2 ), . ..}, and q i denotes the i-th sensor.The sampling periods of multiple sensors are different.τ i is the sampling period of sensor q i .The sensor q i can obtain real values and observed values of the attribute u jk of the fire equipment a j during each sampling period τ i .It can be expressed as . ..g. u true jkði;t i1 Þ denotes real values of sensor q i for measure attribute u jk of fire equipment a j during the first sampling period τ i1 .u false jkði;t i1 Þ denotes observed values of sensor q i for measure attribute u jk of fire equipment a j during the first sampling period τ i1 .These sensors have different sampling periods.At each time step t 0 l , observed value of each sensor is obtained in FERD system.This process is shown in Fig 1 .• Agent Norm N 0 .The norm set N 0 for the agent, which is denoted by N 0 ¼ fr 0 1 ; r 0 2 ; . ..g.Where norm r 0 k is expressed as Objectives of FERD system.The objectives of FERD system are expressed as Eq (5).Minimize the maximum time for FERD system rescue to complete tasks.And all tasks in M are completed.
Constituent of FERD system.The FERD system operates based on the GBBopen platform.In a FERD system, a set of multiple KSs make up a strategy.The CS selects the appropriate KS for the multiple agents to switch.After the strategy is executed, the data on the BB are changed until the fire extinguishing tasks are completed.Certain properties of the agent are obtained from external sensors.The components of the FERD system are described in detail below.
(1) Blackboard.In the FERD system, a task and a type of agent are defined as classes, respectively.Each agent is defined as an instance.The attribute value of each agent is either directly or indirectly defined.
(2) Knowledge Source.KS is a core component of the FERD system.In GBBopen, the attributes of KS are in correspondence with the Agent Norm N 0 , and the properties of the KS are extended, 'postcondition-function'.The relationship between N 0 and KS can be formally described as follows:  e. Evolutionary Strategy.The FERD system provides two evolutionary strategies for autonomously evolving norms: Crossover and Other Method.In Crossover, the expectation of one mutated norm enrich the trigger of another mutated norm.
(3) Control Shell.In the FERD system, after ksa k .exe is executed, ksa k .post is executed.The CS triggers various norms based on event changes to achieve fast switching strategies.

Coordinate transform
Because the three-dimensional coordinate cannot directly transform the two-dimensional coordinate, it is necessary to obtain the mapping relationship of each coordinate.The transformation between the world coordinate and the camera coordinate is through rigid body transformation.The transformation between the camera coordinate and the image coordinate is through perspective projection.The transformation between the image coordinate and the pixel coordinate is through affine transformation.The transforming formula between a point in the world coordinate and a point in pixel coordinate can be expressed as: Among them, Z c indicates the distance between the camera and the point in the world coordinate.This distance can be obtained in other ways.K 3×4 indicates internal reference matrix.RT 4×4 represents external reference matrix.f x , f y indicate focal length.Generally, they are equal.R 3×3 represents rotation matrix, which describes the direction between the coordinate axis of a world coordinate and the coordinate axis of the camera coordinate.t 3×1 indicates translation matrix, which describes the position of the origin of a world coordinate in the camera coordinate.

Camera calibration
The calibration method of Zhengyou Zhang is to fix the world coordinate system on the chessboard.This world coordinate system is defined in advance.Capture an image of the chessboard through the camera.The pixel coordinates (u, v) of each corner point are obtained using the image detection algorithm.The corner point is the connection point of the contour line of the object.The coordinate of each corner point (X w , Y w , Z w =0) is calculated in the world coordinate system.Through Eq 6, the mapping relationship between the world coordinate and pixel coordinate is calculated.The internal parameter matrix (e.g.K 3×4 ) and external parameter matrix (e.g.R 3×3 , t 3×1 ) are obtained.

Monocular measurement
The measurement methods based on monocular vision can be divided into multiple categories.One of the generalizations is the ratio-based approach.The principle is that the distance is inversely proportional to the image size.The calculation formula is: Among them, f represents Focal length; P represents Pixel width of an object, pixel; W expresses Actual width of an object, m; D 0 represents that unknown distance from an object to a camera, m.

Kalman filter based confidence degree
Kalman Filter is a data fusion algorithm.This algorithm fuses the measured values of different sensors with the same object.The measurement noise satisfies a normal distribution.Finally, a more accurate value is obtained.To study the consistency of sensor data, the concept of confidence degree is introduced here.
Suppose there exist n independent sensors.There are q 1 , q 2 , . .., q n .The fusion process of the Kalman Filter based on confidence degree is as follows.
ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ð1 À KÞ * s fuse tÀ 1 q Among them, K indicates Kalman gain.The degree of change in uncertainty after each fusion of data.A lower Kalman gain helps to improve the accuracy of fusion and vice versa.R i represents measurement error of the i-th sensor.R 0 represents measurement error of all sensors.q i t indicates observation data of the i-th sensor.x i t indicates equation of state of the i-th sensor.x 0 i t indicates the optimal estimate of the i-th sensor.It is obtained by classical Kalman filtering.At time t, F i t represents state transfer matrix of the i-th sensor.B i t indicates control matrix of the i-th sensor.m i t represents control vector of the i-th sensor.s i t indicates standard deviation of the i-th sensor.w i t indicates confidence degree of the i-th sensor.q 0 t represents observations based confidence degree.x0 t indicates predicted values based confidence degree.s 0 t indicates standard deviation based confidence degree.q fuse t represents fusion results of multiple sensors.s fuse t indicates standard deviation of multiple sensors.
As can be seen from Eq 8, the effective fusion of multiple sensors is achieved.This process adjusts the confidence degree of each sensor in real time.When the observed values of a sensor is not accurate, the algorithm has a certain fault-tolerant ability.This algorithm affects the overall performance of multi-sensor fusion.In this paper, the Kalman Filter based confidence degree has the ability to be adaptive.

Proposed module
In this paper, for the phenomenon of inaccurate observed values of multiple sensors, Fusion Module is added in FERD system.This section provides a detailed description of the supplement or correct of a single sensor.

Fusion module of FERD system
This study added the fusion module in FERD system.KS consists of three sub-modules.They are the control module, execution module, and fusion module.

KS of fusion module
This subsection provides a detailed description of Fusion Module.Experts have developed a series of fusion strategies for the fusion of multiple sensors.As shown in Fig 2 .Denoising.Interference values exist for every sensor.Interference values are values that describes the scenario inaccurately.In this paper, interference values are collectively referred to as random errors.Then the observed values contains a large number of random errors.The random errors not only increase the computer memory and computation overhead but also increase the computation error.So observed values of each sensor need denoising.The purpose of denoising for sensor q i is to reduce the Standard Deviation σ i .
Spatial alignment.The observed value of each sensor in the same platform using different coordinate systems.The observed value of different sensors involve different coordinate systems.Therefore, observed value must be converted into data in the same coordinate system before fusion.In fact, this process is to calculate the transformation relationship between coordinate systems.The system converts the attribute u jk of the fire equipment a j into a uniform type of sampled data from sensor q i by Spatially Aligning KS.In this paper, the transformation matrix of sensor q i is set to trans i .The observation for each sampling period t 0 l after spatial alignment is u false jkði;t 0 l Þ * trans i .The real value of the sampling period t 0 l after spatial alignment is u true jkði;t 0 l Þ * trans i .Confidence degree update KS.The contribution of each sensor to data fusion is also different.Therefore, this paper proposes the concept of confidence degree.The fusion contribution of each sensor is defined by the confidence degree.For example, sensor q m and q n are both sensors that make observations of u jk .In each sampling period, their confidence degrees are updated by this KS.Their confidence degrees are denoted by w mg and w ng , respectively.Where g ¼ 1; 2; . . .; t max .
Observation filtering.The error of observed values from each sensor is different.Therefore, the concept of error threshold is proposed in this paper.It serves as a precondition for fusion.The error of observed values of sensor q i on u jk in each sampling period is expressed as follows.
For example, sensor q m and q n measure for u jk .Then, this paper calculates the error between observed values and real values.Their errors are denoted as ε mg and ε ng , respectively.Select their error thresholds by this KS.Their error thresholds are denoted as ε m and ε n , respectively.Finally, observed values of two sensors in each sampling period t 0 g are compared with the corresponding error thresholds separately.The filtering operation is performed mainly for Data Fusion KS.
Data fusion.This KS is a process of multi-level, multi-spatial information complement and optimal combination processing by multiple sensors.In this process, multiple sources of data are fully utilized for rationalization and use.This not only takes advantage of multiple sensors operating in concert with each other but also integrates data from other information sources.This improves the intelligence of the whole system, as well as the accuracy and comprehensiveness of the information, and reduces the uncertainty of the information.This KS combines observations from multiple sensors into one data.This operation makes the measured values more accurate.
This paper assumes that there are Γ different fusion algorithms in this module.The observed values of sensor q m and q n for the same attributes (u jk ) are fused.At this point, the observed values from both sensors have been denoised and spatially aligned.During each sampling period t 0 g , FERD system can obtain observed values of all sensors.9ε mg � ε m .Where g ¼ 1; 2; . . .; t max .
By the fusion algorithm γ 2 Γ, the result of Data Fusion KS during each sampling period t 0 g can be expressed as u Among them, the symbol '•' is a symbol for data fusion, which indicates that two observed values are fused.The real values of the same property for the same object is the same for different sensors.The real values of the same property of the same object in each sampling period t 0 l can be expressed as u true g jkðt 0 g Þ .

Fusion decision.
For the results of different data fusion algorithms, the role of this KS is to select the appropriate fusion algorithm γ.This algorithm is a fusion algorithm that is the closest to the real values.That is: Where, j = 1, 2, . .., x; k = 1, 2, . .., y 0 ; γ 2 Γ.

Results
This experiment of performing firefighting tasks in an indoor scenario with multiple obstacles is a specific application scenario of this fusion module proposed.FERD system can use multiple sensors to detect unmanned vehicles during firefighting tasks.To verify the effectiveness of Fusion Module, three sensors are used to assist in the firefighting tasks.This section first introduces experiment design.Then, the experimental steps are described in detail.Subsequently, the experimental results are given.Finally, the effectiveness is further verified by visual analysis.

Experiment design
In this subsection, the experiment design is first described.And the details are described.
Then, the limitations are considered and analyzed.Finally, evaluation criteria are given to verify the effectiveness of this fusion module.Table 3 summarizes the symbols commonly used in this experiment.
Experiment scenario.This experimental sets the experimental scenario as a rectangular environment, as shown in Fig 3.This scenario is a rectangular base station laid by four UWB anchors located at the same horizontal position.One of anchors is used as the origin of rectangular.
Experiment platform and equipment.The experiments in this paper are conducted By taking the Chengdu Kestrel Artificial Intelligence Institute's Interface Description Language (IDL)-Mapping tool (KIS-CORBA) as a cross-language data communication protocol, acquiring real-time information with multiple sensors, and publishing or subscribing to the information through the serial port of the Robot Operating System (ROS) to finally implement GBBopen to control firefighting equipment.This IDL is the IDL of Object Management Group (OMG).
1) Experiment Platform.All experiments require a PC platform and an ARM-embedded platform.The ARM-embedded platform used in this experiment is the Raspberry Pi (RPI).The Python environment (3.7.0), KIS-CORBA, and ROS platforms are installed into the RPI.The Visual Studio environment (VS2017), KIS-CORBA, Lisp environment (Allegro Common Lisp 10.1 express), LinkTrack technology (NoopLoop for NAssistant applications), and GBBopen are installed on the PC.
2) Experiment Equipment.This experiment uses an unmanned vehicle (RoboMaster) as the firefighting equipment for the FERD system, as shown in Fig 4 .i. Sensors.Each piece of unmanned vehicle carried five items, including an IDS, a camera, an UWB anchor, a module of an IMU, and a RPI, as shown in Fig 4 .ii. Application Programming Interface (API).DJI-Innovations (DJI) provides several API instances (https://robomaster-dev.readthedocs.io/zh_CN/latest/)written in Python.This experiment is conducted to realize communication between the unmanned vehicle and GBBopen and to control the wheels of the unmanned vehicle.Because unmanned vehicle is written in Python and GBBopen is written in Lisp, cross-language communication is required.These experiments focused on the communication between Python and Lisp and finally applied the FERD system.
iii.Attributes.Various attributes exist for each firefighting task and each unmanned vehicle.These attributes can be expressed as follows:  The bit pose stored by unmanned vehicle a j in ascending time order, a list, noted as p_lst j p_v The id of the previous of unmanned vehicle a j , noted as p_v j a_v The id of the after of unmanned vehicle, noted as a_v j x,y,z,w Thresholds r The role of unmanned vehicle a j , (leader or follower), noted as r j s The state of unmanned vehicle a j , (stuck or normal), noted as s j fs The following method of unmanned vehicle a j (goal following or trajectory following), noted as fs j tp The destination location (m) of unmanned vehicle a j , noted as tp j tp_lst The destination location stored by unmanned vehicle a j in ascending time order, a list, noted as tp_lst j υ The linear velocity (m/s) of unmanned vehicle a j traveling towards its destination, denoted as υ j ω The angular velocity (rad/s) of unmanned vehicle a j traveling towards its destination, denoted as ω j d The ED (m) of unmanned vehicle a j from tp j to p j , denoted as d j D 1 The ranging of unmanned vehicle a j of p_v j by IDS (m), denoted as The ranging of unmanned vehicle a j of p_v j by Camera (m), denoted as D 2 j w uwb Confidence degree of UWB position Confidence degree of Camera position (IDS ranging) Confidence degree of Camera position (Camera ranging) Confidence degree of Camera position (Camera and IDS fusion ranging) Filters whether UWB position is available for fusion, (0 or 1).

f camera1
Filters whether Camera position (IDS ranging) is available for fusion, (0 or 1).

f camera2
Filters whether Camera position (Camera ranging) is available for fusion, (0 or 1).

f camera3
Filters whether Camera position (Camera and IDS fusion ranging) is available for fusion, (0 or 1).
https://doi.org/10.1371/journal.pone.0287791.t003iv.Data transmission.The remote server is connected to control the unmanned vehicle through Secure Shell (SSH).The position of the unmanned vehicle, the three-axis attitude angle, and the distance from the front are obtained through UWB, IMU, and IDS, respectively.Among them, UWB achieves high-precision indoor positioning by sending ultra-short pulse signals that exploit multipath propagation and the Doppler effect in indoor environments.The unmanned vehicle obtains images through the camera's API and embeds a coordinate transformation program to output coordinates.The ROS serial port then reads the obtained data and publishes it in the form of topics.Finally, they are transmitted to GBBopen through KIS-CORBA.The data of the unmanned vehicle is sent to GBBopen, including pose, velocity, distance, etc. GBBopen sends the data to the unmanned vehicle, such as target point, speed, etc.
v. Pose information.Positioning errors occur due to various factors (e.g., the deployment mode of the base station, obstracles and the weakening of the signal) in the indoor environment.Therefore, the experiment assumes that the information measured by the UWB and IMU is the actual location information of the unmanned vehicle.In Fig 3, the number of unmanned vehicles is 2. The position of two unmanned vehicles are (10.8,1.8) and (15,0.6),respectively.And the position of the firefighting task is (4.2, 0.6).
Experiment contents.In FERD system, the fusion norms are designed, which guide each unmanned vehicle to perform tasks and the fusion positioning of multiple sensors.Briefly introduce the 22 norms that already exist within FERD system.Norm r 1 creates multiple instances of the unmanned vehicle.Norm r 2 and r 3 are rules for setting the switching ways and sorting the fire locations.The switching ways include Command Mode and Automatic Mode.Norm r 4 * r 8 are rules about the following strategy.Among them, Norm r 4 and r 5 are the ways to determine the following of the unmanned vehicle a i .Norms r 7 and r 8 are parameters to set the unmanned vehicle a i following mode.Norm r 9 * r 13 are rules about the Mobilization Strategy.These strategies include Formation, Departure, and Return To the Team.Norm r 12 and r 13 are rules for setting the role of the unmanned vehicle a i .Norm r 14 * r 16 are three rules about the Motion Strategy.Here it is recommended that the linear velocity of the unmanned vehicle is between 0.05 * 0.5m/s.The angular velocity of the unmanned vehicle is between 0 * 90rad/s.Norm r 17 * r 19 are the three rules of the Obstacle Avoidance Strategy.Norm r 20 * r 22 are rules about Evolutionary Strategy.Table 4 shows the added norms and revised norms within FERD system in this paper.Norm r 14 * r 16 are rules about three revisions of the Motion Strategy.Norm r 23 * r 30 are rules about the Following Strategy.Norm r 31 * r 38 are rules about updating confidence degree and filtering of observations.Norm r 39 * r 40 are rules about different algorithms about the fusion of multiple sensors.For example, the fusion algorithm of the Kalman Filter based confidence degree and the weighted average-based fusion algorithm are applied in this paper.Norm r 41 is the rule for fusion decisionmaking.This norm is that the fusion results from different algorithms are compared with gt j .The optimal fusion result is selected.
In indoor scenarios with multiple obstacles, each unmanned vehicle chooses the appropriate norms based on its state and assists in firefighting tasks through the sensors it carries itself.This experiment sets a fixed route for two unmanned vehicles.They pass through (4.8,1.8),(4.8,0.6), and (4.2,0.6) in turn.Eventually reaching the fire location (4.2, 1.2), they perform the fire extinguishing task.
Experiment limitations.The experiment design can verify the effectiveness of the proposed module, but this module has certain limitations (e.g., timeliness).In this experiment, the relative velocity between two unmanned vehicles is set to check the real-time performance.Specifically, the relative velocity of two unmanned vehicles is used as an independent variable.The fusion results of multiple sensors are used as the dependent variable.Finally, the fusion results of multiple sensors are compared with the real values.This experiment sets different relative velocities between two unmanned vehicles to perform the fire extinguishing task.The relative velocities in this experiment are 0.03m/s, 0.06m/s, 0.09m/s, 0.12m/s, 0.15m/s, and 0.2m/s, respectively.
Evaluation metrics.In this experiment, the FER is proposed as the fusion index of this fusion module.GBBopen receives the number of localization coordinates of all sensors simultaneously, expressed by Z.Among them, the distance between the coordinates localized by UWB and the ground truth is large; while the distance between the coordinates localized by other sensors and the ground truth is small.The distance between the coordinates localized by other sensors and the ground truth is small.The number of coordinates of UWB localization is larger than the distance to the ground truth, denoted by z.The FER is expressed as ZÀ z Z × 100%.With effectively fused data, one evaluation index is the error range of ED and MSE between observed values and the ground truth of a single sensor.Another evaluation index is the error range of ED and MSE between the fusion results of multiple sensors and their ground truth.

Steps
First, this experiment analyzes the different sensors in different scenarios.Next, the important parameters inside the norms are trained in indoor scenario with multiple obstacles.Finally, the two unmanned vehicles follow a set fixed route and assist in firefighting tasks.
2) IDS and Camera Ranging.First, the mapping relationship between world coordinates and pixel coordinates is obtained by Formula 6.The internal and external reference matrices of the camera are obtained by camera calibration.Then, a 1 carries Camera and IDS to range unmanned vehicle a 2 .a 2 moves 10 times in sequence at 600mm intervals from 600*6000mm.The distances between a 1 and a 2 are 600mm, 1200mm, 1800 mm, 2400mm, 3000mm, 3600mm, 4200mm, 4800mm, 5400mm, and 6000mm respectively.Finally, the values of the ranging are compared with the real values.
Train error thresholds and confidence degrees.The accuracy and error of the observed values of each sensor are different.Each sensor also contributes differently to the fusion of multiple sensors.Therefore, some parameters of some norms are trained.They include the confidence degree of each sensor and the error threshold of all sensors.In this experiment, the number of training is 10.In the initial stage of training, the confidence degree of each sensor is equal.And the error threshold of all sensors is set to null.The results at the end of the first training are the initial values for the second training.Until the end of the tenth training session, the trained results are the initial values of the unmanned vehicles performing tasks.
Performing tasks.The two unmanned vehicles follow a set fixed route and assist in firefighting tasks through the parameters of the norms that have been trained and the sensors that each carries.

Results and analysis
The results of the experiment steps are given in this paper.Then, the experiment results are analyzed.0.02*6.34m 2 , respectively.This quantitative result illustrates that the observed values of UWB fluctuate more on the y-axis than on the x-axis.The UWB observations fluctuate more in an indoor scenario with multiple obstacles than without obstacles.This is due to the fact that the accuracy of the short edge (y-axis) is worse than the long edge (x-axis) in the unobstructed case.The impact on the ranging accuracy in an indoor scenario with multiple obstacles and the impact of the ranging algorithm can cause the short edge to be affected more than the long edge.

Observed values of different
2) IDS and Camera Ranging.In Figs 7 and 8, there is no data for ranging by IDS in 4800*6000mm, while the data is available for ranging by Camera.The experimental results show that ranging by IDS does not work, while it works by Camera in 4800*6000mm.The experimental results show that observed values of ranging by IDS are smaller than the real values.Table 6 is analyzed from the quantitative perspective as follows.In an indoor scenario without obstacles, the error range between ranging by IDS and real values is 134.86*420.35mm.The SD of these errors is

Real values
Error in an indoor scene without no obstacle Error in an indoor scene with multiple obstacles

IDS and Real values (mm) Camera and Real values (mm) IDS and Real values (mm) Camera and Real values (mm)
111.70.The range of error between ranging by Camera and real values is -1334.713*489.13mm.The SD of these errors is 489.13.The SD of the Camera is 4.38 times higher than the IDS.In an indoor scenario with multiple obstacles, the error range between ranging by IDS and real values is 115.74*647.74.The SD of these errors is 202.34.The range of error between ranging Camera and real values is -1154.43*268.55mm.The SD of these errors is 529.73.The SD of ranging by Camera is 2.62 times higher than ranging by IDS.These results show that ranging by IDS and Camera has almost no effect in different scenarios.However, ranging by IDS is more stable than ranging by Camera.This is due to the detection of the target by a Camera that has errors (i.e., errors in the pixel width), resulting in errors in the numerator of Formula (7).
3) UWB, IDS and Camera Positioning.Tables 7 and 8 shows the data analysis between the observed values and real values by different sensors in different scenarios.In these table, vertical represents real values, horizontal indicates the error between observed values and real values by different sensors.
In Figs 9 and 10, the trend of red line is flat, while the trends of pink, purple, and green lines are fluctuation.The trend of pink line are greater than purple and green lines.When the real coordinates of a 2 (8.4,1.8) and a 1 are (3.6,1.8),(3,1.78) and (2.4,1.82) in that order, there is no data for ranging by IDS.So no observed values are Camera position (IDS ranging).While the data is available for ranging by Camera, so observed values are Camera position (Camera ranging).In Tables 7 and 8, this experiment compares the observed values of a 1 by different   From the above qualitative and quantitative analysis results, it is clear that the stability of UWB observations varies from scenario to scenario.This experiment allows the IDS and Camera to complement or fuse each other for ranging the unmanned vehicle.When the Camera (IDS ranging) measurement fails, Camera position (Camera ranging) and UWB can complement each other.
Train error thresholds and confidence degrees.Timeliness.Table 9 shows the timeliness results of the fusion module.The table indicates the relative speeds of the two unmanned vehicles in the horizontal direction.The longitudinal direction indicates whether the position after fusion is lagged and the lag value.It is obvious  from the table that there is no lag in the fusion results of multiple sensors when the relative speeds between a 1 and a 2 are set at 0.03m/s, 0.06m/s, 0.09m/s, and 0.12m/s.While the relative speed between a 1 and a 2 is 0.15m/s and 0.2m/s, the fusion results of multiple sensors show lag.At a relative velocity of 0.15m/s, the fusion results differ from the real values by 0.3m.At a relative velocity of 0.2m/s, the fusion results differ from the real values by 2m.From the analysis in the table, it is clear that the relative velocity is within 0.12m/s to ensure the validity of the fusion results of multiple sensors.
Performing tasks.shows the visualization results of the FERD system performing task.a 1 and a 2 perform firefighting tasks in an indoor scenario with multiple obstacles.Horizontal axis indicates the length of the rectangle.Vertical axis indicates the width of the rectangle.
In Fig 13, a 1 follow a fixed route.It starts at (10.8,1.8) and passes through J 1 , J 2 and J 3 in sequence.Finally, it reaches fire T (4.2,1.2) to extinguish the fire.In fact, it is unavoidable that a 1 's wheels have a slight deviation when moving forward.As a 1 moves forward, simultaneously and quickly adjust its deviation, so that a 1 does not differ too much from this route.It is equivalent to a 1 moving along a track.Therefore, the real trajectory of a 1 coincides with the fixed route.The red line indicates the real trajectory of a 1 .a 2 starts from (15,0.6). a 2 passes through A 0 , B 0 , C 0 , and D 0 in that order.a 2 finally reaches (4.05,1.08).The blue line indicates the real trajectory of a 2 .The trajectories of a 1 and a 2 are from right to left.The yellow line indicates the a 2 by multiple sensors to observe a 1 .The yellow line is also a fusion trajectory fused of a 1 .
However, the positioning trajectory obtained by UWB is not consistent with the real trajectory in Figs 13 and 14.This is due to the fact that the experiment is conducted in an indoor environment with many obstacles.The accuracy of UWB positioning is affected by signal interference, reflections, and other factors.In this paper, Fig 13 is analyzed from different perspectives.First, compare the distances of a 1 and a 2 to J 1 .It can be seen that the ED from a 1 to J 1 is shorter than a 2 to J 1 .From the above analysis, a 1 is the leader and a 2 is the follower.From the figure, it can be seen that a 1 gradually approaches J 1 .According to the Norm r 4 , then a 2 uses the goal following approach toward a 1 .In this figure, a 2 by multiple sensors can observe a 1 when a 1 has traveled to A. Meanwhile, a 2 has traveled to A 0 .According to the Norm r 25 *r 27 , so a 2 takes a goal following towards a 1 .Until a 2 by multiple sensors do not observe a 1 , at this point a 1 travels to B. Meanwhile, a 2 has traveled to B 0 .Since both a 1 and a 2 are in the execution task area and a 2 by multiple sensors do not observe a 1 .According to Norm r 5 , then a 2 takes a trajectory following towards a 1 .a 2 travels from B 0 to C 0 .Similarly, a 2 by multiple sensors to observe a 1 when a 1 has traveled to C. Meanwhile, a 2 has traveled to C 0 .According to the Norm r 25 *r 27 , so a 2 takes a goal following towards a 1 .Until a 2 by multiple sensors do not observe a 1 , at which point a 1 travels to D. a 2 has traveled to D 0 .Since both a 1 and a 2 are in the execution task area and a 2 by multiple sensors do not observe a 1 .According to Norm r 5 , then a 2 takes a trajectory following towards a 1 .a 2 starts traveling from D 0 .Until both a 1 and a 2 complete their firefighting tasks.
Evaluation.This paper statistics the raw data acquired by GBBopen.GBBopen receives the number of localization coordinates of all sensors simultaneously is 4070.The number of coordinates of UWB localization that are larger than the distance to the ground truth is 3608.So FER is calculated as 11.35%.Table 10 shows compare results of the range of error and MSE between single sensor and multi-sensor fusion.The range of error of ED between the single sensor observation and the true value is 0.80*1.47m.The error range of ED between the observed and true values after multi-sensor fusion is 0.59*1.02m.The Min of the error of the ED after multi-sensor fusion increased by 26.25% over the single-sensor observations.The Max of the error of the ED after multi-sensor fusion decreased by 30.61% over the single-sensor observations.The range of MSE between the single sensor and real values of the a 1 after multi-sensor fusion is 1.06*1.35m 2 .The MSE of the multi-sensor fusion results over the real value of a 1 is 0.66*0.86m 2 .The Min of the MSE after multi-sensor fusion is 37.74% higher than that of the single-sensor observations.The Max of the MSE after multi-sensor fusion is increased by 36.30% compared to that of the single-sensor observation.
The analysis of the above experiment results shows that the fusion of multiple sensors is achieved in this paper.This fusion module ensures precise positioning between adjacent unmanned vehicles using multiple sensors.The unmanned vehicles are coordinated to accomplish the firefighting task.This fusion module reduces the impact of FERD system in open environments.This module also improves the environmental awareness of each unmanned vehicle.

Discussion
Norms are designed by experts in multiple fields.Each expert can describe different fields according to his experience.This fusion module involves multiple domains.Therefore, experts construct complex fusion strategies in a data processing module by combining multiple norms.When the measured data is unstable, this fusion module can reflect a certain degree of robustness.For example, if a sensor is not working or information is confusing, the FERD system can continue to perform firefighting tasks.The FERD system demonstrates its scalability when new needs arise.For instance, this paper adds new norms within this fusion module.These norms can meet the needs of subsequent environments, and long-lasting applications.
This fusion module proposed in this paper suffers from certain limitations.This paper discuss the uncertainties and limitations of fusion module in terms of the delay of the data, the suitability of the sensors, and the protection measures of the sensors.
• Data latency.The data acquisition is performed in FERD system with ROS as the medium.
The first consideration is how to ensure real-time data fusion.The involved delay includes transmission delay, propagation delay, processing delay, etc.So it is more demanding on the network.FERD system is an event-trigger mechanism.It is more important the specific events monitored change at any time.This mechanism has strong random characteristics.This mechanism cannot predict the target object.In some respects, this may limit the open system.However, in a dynamic environment, this is more generic for fusion methods.This generalization is the fusion of data by norms.• Sensor selection.Fire environments have special properties, such as high temperatures and dense smoke.These special properties can have an impact on the performance and reliability of the sensor.The range of applications for a sensor can also affect its performance and reliability in a fire environment.Namely, some sensors may only be suitable for specific types of fires and may not work properly for other types of fires.Therefore, the range of application of sensors needs to be considered when applying them to avoid limitations and misjudgments.The sensors covered in this paper are a range of DJI-Innovations RoboMaster (EP) products, including an infrared depth sensor (IDS) and a camera.IDS has an operating temperature range of -10 to 40 degrees Celsius.It is possible for the range accuracy to be reduced or even for the measurement to fail in special weather or environments, such as dense smoke, rain, fog, or direct sunlight.The camera needs to ensure that it is not disturbed and is unobstructed.There are video streams and images in the thick smoke.Its operating temperature range is -10 to 40 degrees Celsius.Extreme environments are not considered in this paper.For instance, the fire environment when studying evacuation strategies for large public buildings [25].Therefore, the sensors involved cannot face complex high-temperature or dense smoke environments and are only suitable for mild fire situations.This paper also proposes some methods to improve the data collection capability of the FERD system and ultimately improve fusion efficiency.For example, sensors that can resist harsh environments could replace the above sensors to further improve their performance and stability.
• Sensor protection.Protective measures for sensors in a fire environment are also very important.For example, packaging and encapsulation with fire-resistant materials are required to protect the sensors from high temperatures and smoke.Moreover, the protection level of the sensor and the match between the protection level and the application environment need to be considered to ensure the performance and reliability of the sensor.
At present, many possible applications are being considered to test the feasibility of this fusion module.To move from the current system to the actual application, it is also necessary to add some functions reflecting the real situation in these norms (e.g. the important parameters of some norms).In theory, these parameters use constant values.In application, these parameters use empirical values.These experience values are dynamically adjustable.Also, the parameters can be affected by random, system noise, due to larger environmental factors.

Conclusion
This paper proposes a fusion module in the FERD system based on Blackboard Architecture.The main goal of this fusion module is to overcome the inaccuracy of a single sensor in indoor scenarios with multiple obstacles and improve measurement accuracy.The module utilizes multiple sensors to complement or correct the positioning of each unmanned vehicle.Specifically, this module develops a variety of fusion-related processing techniques for precise positioning.This paper then proceeds to gain experience data on the confidence degree, error of different sensors, and timeliness of this module by training in an indoor scenario with multiple obstacles.Compared with a single sensor, the module proposed in this paper is switchable and scalable.Finally, the performance of the fusion module was evaluated to demonstrate the effectiveness of localization based on the field sensors.Some important applications are target identification and target tracking tasks for multiple agents.These areas mainly include identifying the motion intention of the target, local path planning, etc.For example, when a sensor has short-term abnormal data, the system determines the abnormal conditions based on the status of the abnormal data and automatically filters this abnormal data.Then this system can still work normally without affecting subsequent operations.In the future, we will apply the proposed fusion module to other applications.The application of autonomous driving systems, for example, can improve safety.

2 )
post.KSs are divided into two submodules: the Switch Module and Strategy Module. 1) Classification of Switch Module.The agent can switch between the strategies based on multiple switch modules.The control modes are divided into Automatic Mode and Command Mode.Multiple agents can switch strategies based on their own identification or environmental identification, or based on user requirements or external guidance.Classification of Strategy Module.Different strategies are generated from the agent state and the collected environmental information.These mainly include the Following Strategy, Motor Strategy, Evolution Strategy, Obstacle Avoidance Strategy, and Mobilization Strategy.

Fig 5 .Fig 6 .
Fig 5.In indoor scenario without no obstacle.Comparison of UWB position and real values in different scenarios.https://doi.org/10.1371/journal.pone.0287791.g005 Figs 7 and 8 show the results of the observed values and real values analyzed by qualitative.The real values are distance between a 1 and a 2 .The observed values are a 1 ranging a 2 through the carried IDS and the Camera.Horizontal axis indicates the number of experiments.Vertical axis indicates the observed values of ranging.The red line

Fig 7 .
Fig 7.In indoor scenario without no obstacle.Comparison of observed values and real values of ranging by different sensors in different scenarios.https://doi.org/10.1371/journal.pone.0287791.g007

Fig 8 .
Fig 8.In indoor scenario with multiple obstacles.Comparison of observed values and real values of ranging by different sensors in different scenarios.https://doi.org/10.1371/journal.pone.0287791.g008 Figs 9 and 10 show the results of the observed values and real values analyzed by qualitative.Horizontal axis indicates the length of the rectangle.Vertical axis indicates the width of the rectangle.The unmanned vehicle a 2 is compared ten times in order from right to left.The red line indicates the real values of a 2 .The pink line indicates the observed values of a 1 by UWB to position itself.The purple line indicates that a 1 by IDS to range a 2 .At the same time, the observed values of a 1 by Camera to position a 2 .The green line indicates the monocular range results of a 1 by Camera to a 2 .Also, the observed values of a 1 by Camera to position a 2 .

Fig 10 .
Fig 10.In indoor scenario with multiple obstacles.Comparison of observed values and real values in different scenarios.https://doi.org/10.1371/journal.pone.0287791.g010 Figs 11 and 12 show the change curve of training confidence degree and error threshold.

Fig 11 .
Fig 11.Training confidence.Training variation curves of confidence degree.The different color lines show the change training trend of different sensors' confidence degrees.The pink line shows the change training trend of UWB positioning.The purple line shows the change training trend of Camera position (IDS ranging).The green line shows the change training trend of Camera position (Camera ranging).The yellow line indicates the change training trend of Camera position (Camera and IDS fusion ranging).https://doi.org/10.1371/journal.pone.0287791.g011

Fig 13 .
Fig 13.Real trajectory of a 1 and a 2 .This figure shows the real trajectory of a 1 and a 2 are perform a task.https://doi.org/10.1371/journal.pone.0287791.g013 Fig 15 shows the valid coordinates of a 1 localized by different sensors.The figure includes the visualization results of single-sensor and multi-sensor fusion.The red dots are the coordinates of a 1 localized by the UWB carried by a 1 .Purple dots are the coordinates localized by the fusion module for a 1 .Specifically, a 1 is fused through the positioning coordinates of a single sensor and a 2 through multiple sensors observing the positioning coordinates of a 1 .In addition, the Fig 15 is somewhat different from the result of locating a 1 by fusion in Fig 14.The trajectory of a 1 by UWB localization in Fig 14 is a continuous line, as shown in the red line in Fig 14.While it is a discontinuous point, as shown in the red dots in Fig 15.This is due to the fact that the coordinates obtained by GBBopen are different from the frequency of the localization coordinates obtained by UWB.Among them, UWB obtains the positioning coordinates more frequently than the coordinates obtained by GBBopen.And the data obtained by GBBopen is also susceptible to the influence of net clusters.

Table 2 . Recording of instantaneous events and states.
If an instantaneous event e i 2 E occurs, the ASTS data are read by the transient state s j 2 S. iv.Judge Tri k of Norm N. v.If Tri k = True, Norm r k is triggered; perform Act k and judge Exp k .vi.The results for Act k and Exp k are restored as Res Act and Res Exp , respectively.vii.s i , e j , r k , Res Act , Res Exp denote the restored paths.

Table 3 .
Symbol notations.Camera and IDS fusion ranging) of unmanned vehicle a j , noted as pcamera3 j pFinal position of unmanned vehicle a j by multi-sensor fusion, noted as p j p_lst